Image processing apparatus and image processing method

ABSTRACT

State information corresponding to a subject on a visible light image, such as temperature information and sound field information, is acquired from state information indicating a subject state not indicated on the visible light image by an information acquisition unit. An effect processing unit performs an effect process on the visible light image on the basis of the state information acquired by the information acquisition unit. In the effect process, an effect component image is superimposed or a subject image is modified based on the type and temperature of the subject and an effect component image is superimposed or a subject image is modified based on the type of a sound source and the volume of a sound.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Phase of International PatentApplication No. PCT/JP2017/000254 filed on Jan. 6, 2017, which claimspriority benefit of Japanese Patent Application No. JP 2016-053609 filedin the Japan Patent Office on Mar. 17, 2016. Each of theabove-referenced applications is hereby incorporated herein by referencein its entirety.

TECHNICAL FIELD

This technology relates to an image processing apparatus, an imageprocessing method, and a program and makes it possible to improve thereality of a visible light image.

BACKGROUND ART

Conventionally, a visible light image is acquired using an imagingapparatus to create a preferable photograph by adjusting the contrast,brightness, white balance, hue, saturation and the like of the acquiredvisible light image.

In addition, an alarm or the like is made on the basis of stateinformation indicating a subject state not indicated on the visiblelight image. For example, Patent Document 1 discloses that imaging isperformed using an infrared imaging apparatus such that temperatureinformation on a road surface is acquired as the state informationindicating a subject state and, on the basis of the acquired temperatureinformation, a region estimated to be frozen in a road surface image isemphasized to be displayed.

Meanwhile, according to Non-Patent Document 1, in accordance with atemperature change caused on a screen by a person, the saturation, hue,three-dimensional shape, and the like of an image projected on thisscreen are changed.

CITATION LIST Patent Document

-   Patent Document 1: Japanese Patent Application Laid-Open No.    2009-248665

Non-Patent Document

-   Non-Patent Document 1: Daisuke Iwai, Kosuke Sato “ThermoRetouch &    ThermoModeler: Heat Sensation in Image Creation with Thermal    Vision”, The Journal of the Society for Art and Science, Vol. 6, No.    2 (2007): pp. 65-75

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

Incidentally, visible light images visually having no difference areregarded as the same as each other even if subjects are, for example, ina high temperature state or a low temperature state and its reality ispoor. In addition, Patent Document 1 is a technology for making an alarmor the like on the basis of the state information and it is thus notpossible to improve the reality of the visible light image. Furthermore,Non-Patent Technical Document 1 is a technology for changing an image inaccordance with a temperature change caused on a screen by a person andthe reality of the visible light image cannot be improved because achange in an image does not correlate with a subject state.

Thus, it is an object of this technology to provide an image processingapparatus, an image processing method, and a program capable ofimproving the reality of a visible light image.

Solutions to Problems

A first aspect of this technology is an image processing apparatusincluding:

an information acquisition unit that acquires state informationcorresponding to a subject on a visible light image from stateinformation indicating a subject state not indicated on the visiblelight image; and

an effect processing unit that performs an effect process on the visiblelight image on the basis of the state information acquired by theinformation acquisition unit.

In this technology, state information corresponding to a subject on avisible light image, such as temperature information and sound fieldinformation, is acquired from state information indicating a subjectstate not indicated on the visible light image by the informationacquisition unit. In the effect processing unit, an effect process basedon the temperature information and the sound field information isperformed on the visible light image on the basis of the stateinformation acquired by the information acquisition unit.

The information acquisition unit acquires type information on thesubject by performing a recognition process for the subject, and theeffect processing unit superimposes an effect component image accordingto a temperature and a type of the subject onto a subject image on thebasis of state information and the type information. In addition, anattribute of the effect component image is controlled in accordance witha difference in temperature. The effect component image is employed asan image schematically representing a visually observable phenomenonthat occurs in accordance with temperature. In addition, the effectprocessing unit modifies the subject image on the visible light imageinto an image to which a change in a physical state of the subject, forexample, a change between a solid phase and a liquid phase, is added inaccordance with a temperature of the subject. The effect processing unitcontrols a modification level in accordance with a difference in thetemperature. In addition, the effect processing unit may perform coloradjustment on the visible light image in accordance with a temperatureof the subject in such a manner that color adjustment to a warm colorsystem is performed when a temperature of the subject is high and coloradjustment to a cool color system is performed in a case where atemperature of the subject is low. In a case where the state informationis the sound field information, the effect processing unit superimposesan effect component image associated with a sound output from a soundsource onto a subject on the visible light image corresponding to aposition of the sound source based on the sound field information on aregion of the visible light image. In addition, the effect processingunit adjusts an effect amount for the visible light image in accordancewith intensity of a sound output from a sound source.

A second aspect of this technology is an image processing methodincluding:

acquiring, by an information acquisition unit, state informationcorresponding to a subject on a visible light image from stateinformation indicating a subject state not indicated on the visiblelight image; and

performing, by an effect processing unit, an effect process on thevisible light image on the basis of the state information acquired bythe information acquisition unit.

A third aspect of this technology is a program that causes a computer toimplement:

a function of acquiring state information corresponding to a subject ona visible light image from state information indicating a subject statenot indicated on the visible light image; and

a function of performing an effect process on the visible light image onthe basis of the state information corresponding to the subject on thevisible light image.

Note that the program according to the present technology is a programthat can be provided, for example, to a general-purpose computer capableof executing a variety of program codes by a storage medium or acommunication medium that provides a program in a computer-readableformat, for example, a storage medium such as an optical disc, amagnetic disk, and a semiconductor memory or a communication medium suchas a network. By providing such a program in a computer-readable format,a process according to the program is implemented on the computer.

Effects of the Invention

According to this technology, state information corresponding to asubject on a visible light image is acquired from state informationindicating a subject state not indicated on the visible light image bythe information acquisition unit. The effect processing unit performs aneffect process on the visible light image on the basis of the stateinformation acquired by the information acquisition unit. Therefore, thereality of the visible light image can be improved. Note that theeffects described in the present description merely serve as examplesand not construed to be limited. There may be an additional effect aswell.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram exemplifying the configuration of a system using animage processing apparatus.

FIG. 2 is a diagram exemplifying the configuration of an imageprocessing apparatus of a first embodiment.

FIG. 3 is a flowchart illustrating the operation of the firstembodiment.

FIGS. 4A, 4B, and 4C are diagrams illustrating an operation example ofthe first embodiment.

FIGS. 5A, 5B, 5C, 5D, 5E, and 5F are diagrams for explaining anoperation of generating an effect component image.

FIG. 6 is a diagram exemplifying the configuration of an imageprocessing apparatus of a second embodiment.

FIG. 7 is a flowchart illustrating the operation of the secondembodiment.

FIGS. 8A and 8B are diagrams illustrating a case where the temperatureis high as an operation example of the second embodiment.

FIGS. 9A and 9B are diagrams illustrating a case where the temperatureis low as an operation example of the second embodiment.

FIG. 10 is a diagram exemplifying the configuration of an imageprocessing apparatus of a third embodiment.

FIGS. 11A and 11B are diagrams illustrating a case where the temperatureis low as an operation example of the third embodiment.

FIGS. 12A and 12B are diagrams illustrating a case where the temperatureis high as an operation example of the third embodiment.

FIG. 13 is a diagram exemplifying the configuration of a fourthembodiment.

FIGS. 14A and 14B are diagrams exemplifying a visible light image inwhich the reality is improved using the effect component image.

FIGS. 15A and 15B are diagrams exemplifying a visible light image inwhich the reality is improved using the effect component image andmodification control information.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, modes for carrying out the present technology will bedescribed. Note that the description will be given in the followingorder.

1. Configuration of Image Processing Apparatus

2. First Embodiment

2-1. Configuration of First Embodiment

2-2. Operation of First Embodiment

3. Second Embodiment

3-1. Configuration of Second Embodiment

3-2. Operation of Second Embodiment

4. Third Embodiment

4-1. Configuration of Third Embodiment

4-2. Operation of Third Embodiment

5. Fourth Embodiment

5-1. Configuration of Fourth Embodiment

5-2. Operation of Fourth Embodiment

6. Other Embodiments

1. Configuration of Image Processing Apparatus

FIG. 1 exemplifies the configuration of a system using an imageprocessing apparatus. The system 10 has a visible light image generationunit 11, a state information generation unit 12, and an image processingapparatus 20. The visible light image generation unit 11 is an apparatusthat generates image data of a visible light image and, for example, animaging apparatus that has sensitivity in a visible light region and isconfigured using a color filter, an imaging device, or the like is used.The visible light image generation unit 11 generates image data of avisible light image including a subject and outputs the generated imagedata to the image processing apparatus 20. The state informationgeneration unit 12 is an apparatus that generates state informationindicating a subject state not indicated on the visible light image. Forexample, in a case where temperature information is generated as thestate information, an infrared imaging apparatus or the like that hassensitivity in a far infrared light region and is configured using amicrobolometer or the like is used as the state information generationunit 12. Meanwhile, for example, in a case where sound field informationis generated as the state information, a microphone array apparatus orthe like in which a plurality of microphones is, for example,two-dimensionally arranged is used as the state information generationunit 12. The state information generation unit 12 generates the stateinformation indicating a subject state not indicated on the visiblelight image and outputs the generated state information to the imageprocessing apparatus 20.

On the basis of the state information generated by the state informationgeneration unit 12, the image processing apparatus 20 performs an effectprocess on the visible light image of the image data generated by thevisible light image generation unit 11, to improve the reality on thebasis of the state information generated by the state informationgeneration unit 12. The image processing apparatus 20 has an informationacquisition unit 21 and an effect processing unit 22. The informationacquisition unit 21 acquires state information corresponding to asubject on the visible light image from the state information generatedby the state information generation unit 12. The effect processing unit22 performs the effect process on the visible light image on the basisof the state information acquired by the information acquisition unit21.

2. First Embodiment

In a first embodiment, the state information is the temperatureinformation and the image processing apparatus improves the reality ofthe visible light image by superimposing an effect component imageaccording to the temperature of the subject onto the visible lightimage.

2-1. Configuration of First Embodiment

FIG. 2 exemplifies the configuration of an image processing apparatus ofthe first embodiment. The image processing apparatus 20-1 has aninformation acquisition unit 21-1 and an effect processing unit 22-1.The effect processing unit 22-1 has an effect component image generationunit 221 and an effect component image superimposition unit 222.

The information acquisition unit 21-1 performs a recognition process anda region segmentation process for the subject on the basis of thevisible light image. The information acquisition unit 21-1 identifieswhat type of entity the subject is in the recognition process andgenerates type information. The information acquisition unit 21-1 alsoidentifies the region of the subject in the region segmentation process.As for the recognition process and the region segmentation process, forexample, it is possible to identify the type and the region of thesubject using the technology disclosed in the document “Shotton, Jamie,et al., “TextonBoost: Joint appearance, shape and context modeling formulti-class object recognition and segmentation”, Computer Vision ECCV2006, Springer Berlin Heidelberg, 2006, pp. 1-15”. In addition, theinformation acquisition unit 21-1 acquires the temperature informationon each subject from the temperature information, which is the stateinformation generated by the state information generation unit 12, onthe basis of a processing result of the region segmentation process.Furthermore, the information acquisition unit 21-1 may confirm theconsistency between the temperature information and the typeidentification result. By confirming the consistency, for example, evenif a person is erroneously identified as a tree, it can be detected thatthe subject is not a tree on the basis of the temperature information,such that the accuracy of the recognition process can be enhanced. Theinformation acquisition unit 21-1 acquires the type information on thesubject and its region and temperature information on the visible lightimage for each subject to output to the effect component imagegeneration unit 221 of the effect processing unit 22-1.

The effect component image generation unit 221 generates an effectcomponent image for improving the reality in accordance with the typeand temperature of the subject on the basis of the type information andtemperature information on the subject. The effect component image is animage schematically representing a visually observable phenomenon thatoccurs in a subject in accordance with temperature. For example, in acase where the subject is a drink container, steam is produced above thedrink container when the liquid in the drink container is at hightemperature, while water droplets are produced around the drinkcontainer when the liquid is at low temperature. Accordingly, the effectcomponent image generation unit 221 generates an effect component imagefor the drink container using pre-generated element images individuallyindicating, for example, steam, water droplets, and the like. Inaddition, the effect component image generation unit 221 may controlattributes of the effect component image, for example, the dimension andthe number of the element images, on the basis of the temperatureinformation. For example, as the temperature rises, the effect componentimage generation unit 221 enlarges the size of an element imageindicating steam or increases the number of the element images.Alternatively, as the temperature lowers, the number of element imagesindicating water droplets may be increased. The effect component imagegeneration unit 221 outputs the generated effect component image to theeffect component image superimposition unit 222. Note that the effectcomponent image generation unit 221 may store in advance the effectcomponent images according to the types and temperatures of subjectsand, on the basis of the type information and the temperatureinformation supplied from the information acquisition unit 21-1, selectsan effect component image corresponding to the type and the temperatureto output to the effect component image superimposition unit 222.

The effect component image superimposition unit 222 superimposes theeffect component image generated by the effect component imagegeneration unit 221 onto the image of the corresponding subject, therebyimproving the reality of the visible light image. The image region ofthe subject is identified by the region segmentation process describedabove. In addition, the effect component image is generated inaccordance with the type and temperature of the subject whose imageregion has been identified. Accordingly, the effect component imagesuperimposition unit 222 can superimpose the effect component image ontothe image of the corresponding subject. Note that the superimposition ofthe effect component image onto the image of the subject is notrestricted to the case of superimposing the effect component image ontop of the image of the subject but also includes the case ofsuperimposing the effect component image onto a position specified usingthe image of the subject as a reference. For example, an effectcomponent image indicating water droplets is superimposed on top of theimage of the drink container. Meanwhile, an effect component imageindicating steam is superimposed at a position above the image of thedrink container using the image of the drink container as a reference.In addition, the effect component image superimposition unit 222 cansuperimpose an effect component image of an appropriate size onto theimage of the subject if the size of the effect component image isadjusted in accordance with the size of the image region of the subject.Alternatively, the effect component image may be generated by the effectcomponent image generation unit 221 in accordance with the size of theimage region of the subject. The effect component image superimpositionunit 222 records the visible light image, onto which the effectcomponent image has been superimposed to improve the reality, to arecording medium or outputs the visible light image to externalequipment such as a display apparatus or a recording apparatus.

2-2. Operation of First Embodiment

FIG. 3 is a flowchart illustrating the operation of the firstembodiment. In step ST1, the effect processing unit 22-1 initializes aparameter value i indicating the subject on which the effect process isto be performed and specifies the parameter value i as “0” to proceed tostep ST2.

In step ST2, the effect processing unit 22-1 sets the number of subjectsto the number of objects N. The effect processing unit 22-1 sets thenumber of subjects identified by performing the recognition process andthe region segmentation process in the information acquisition unit 21-1to the number of objects N and proceeds to step ST3.

In step ST3, the effect processing unit 22-1 initializes the effectcomponent image to bring about a state in which an effect componentimage to be superimposed on the visible light image is not generated andproceeds to step ST4.

In step ST4, the effect processing unit 22-1 identifies whether theparameter value i is smaller than the number of objects N. In a casewhere the parameter value i is smaller than the number of objects N, theeffect processing unit 22-1 proceeds to step ST5 and, in a case wherethe parameter value i is equal to or larger than the number of objectsN, the effect processing unit 22-1 terminates the process.

In step ST5, the effect processing unit 22-1 selects element imagecandidates. The effect component image generation unit 221 of the effectprocessing unit 22-1 selects element image candidates corresponding tothe type of an i-th subject and proceeds to step ST6.

In step ST6, the effect processing unit 22-1 selects an element imagefrom the candidates. The effect component image generation unit 221 ofthe effect processing unit 22-1 selects an element image to be used forgenerating the effect component image from the element image candidatesselected in step ST5 on the basis of the temperature information on thei-th subject and proceeds to step ST7.

In step ST7, the effect processing unit 22-1 generates the effectcomponent image. The effect component image generation unit 221 of theeffect processing unit 22-1 adjusts the image size and/or the number ofthe element images selected in step ST6 on the basis of the temperatureinformation on the i-th subject. In addition, the effect component imagegeneration unit 221 generates the effect component image to besuperimposed onto the image of the i-th subject using the adjusted imagesize and number of the element images and proceeds to step ST8.

In step ST8, the effect processing unit 22-1 superimposes the effectcomponent image. The effect component image superimposition unit 222 ofthe effect processing unit 22-1 superimposes the effect component imagegenerated in step ST7 onto the image of the i-th subject on the visiblelight image and proceeds to step ST9.

In step ST9, the effect processing unit 22-1 updates the parameter valuei to specify the parameter value i as “i=i+1” and returns to step ST4.

The effect processing unit 22-1 performs the process in the flowchartillustrated in FIG. 3 to perform the process of generating the effectcomponent image and the process of superimposing the effect componentimage onto the visible light image for each subject identified by theinformation acquisition unit 21-1 and generates a visible light imagewith improved reality.

FIGS. 4A, 4B, and 4C illustrate an operation example of the firstembodiment. In addition, FIGS. 5A, 5B, 5C, 5D, 5E, and, 5F are diagramsfor explaining an operation of generating the effect component image.

In FIG. 4A, the visible light image based on the image data output fromthe visible light image generation unit 11 is exemplified and, in FIG.4B, a temperature image based on the state information output from thestate information generation unit 12 is exemplified. The temperatureimage indicates that the temperature rises as the luminance becomeshigher and the temperature lowers as the luminance becomes lower. It isthus possible to identify the subjects on the visible light image fromthe temperature image in such a manner that, for example, a coffee cupOBa is at high temperature and a beer glass OBb is at low temperature.

In FIG. 5A, element images registered in advance are exemplified. Theelement image schematically represents a visually observable phenomenonthat occurs in accordance with temperature and, for example, imagesschematically representing steam produced at high temperature, waterdroplets produced at low temperature, body tremor produced at lowtemperature, and the like are registered.

In FIG. 5B, element images selected as candidates are exemplified. In acase where the subject is a drink container such as a coffee cup or abeer glass, the effect component image generation unit 221 selectselement images schematically representing steam and water dropletsproduced in the drink container as candidates.

In FIG. 5C, an element image selected from the candidates on the basisof the temperature information on the coffee cup is illustrated. Asillustrated in FIG. 4B, the temperature information indicates that thecoffee cup OBa is at high temperature. Accordingly, since the coffee cupOBa is at high temperature, the effect component image generation unit221 selects an element image schematically representing steam.Furthermore, on the basis of the temperature information on the coffeecup OBa, for example, the effect component image generation unit 221enlarges the size of the element image indicating steam and increasesthe number of the element images as the temperature of the coffee cupOBa rises and consequently, generates an effect component image EOBaillustrated in FIG. 5D.

In FIG. 5E, an element image selected from the candidates on the basisof the temperature information on the beer glass is illustrated. Asillustrated in FIG. 4B, the temperature information indicates that thebeer glass OBb is at low temperature. Accordingly, since the beer glassOBb is at low temperature, the effect component image generation unit221 selects an element image schematically representing a water droplet.Furthermore, on the basis of the temperature information on the beerglass OBb, for example, the effect component image generation unit 221increases the number of the element images indicating a water droplet asthe temperature of the beer glass OBb lowers and consequently, generatesan effect component image EOBb for the beer glass OBb illustrated inFIG. 5F.

The effect component image superimposition unit 222 superimposes theeffect component image generated in accordance with the type informationand the temperature information on the subject onto the correspondingsubject image on the visible light image with a picture sizecorresponding to the region size of the subject. Accordingly, asillustrated in FIG. 4C, the effect component image EOBa indicating steamis superimposed onto the image of the coffee cup OBa at an appropriateposition in an appropriate size. Meanwhile, the effect component imageEOBb indicating water droplets is superimposed onto the image of thebeer glass OBb at an appropriate position in an appropriate size.Therefore, it is possible to output a visible light image with improvedreality from the image processing apparatus 20-1.

As described thus far, according to the first embodiment, the effectcomponent image generated in accordance with the type and temperature ofthe subject is superimposed onto the image of the corresponding subjectin regard to the images of the subjects on the visible light image.Accordingly, it is possible to generate a visible light image having thereality, in which the imaging object and the imaging situation arereflected. In addition, effects are added on the basis of the actualtemperature information on the subject, such that it is possible tomaintain a natural influence of the effects.

3. Second Embodiment

In a second embodiment, the state information is the temperatureinformation and the image processing apparatus improves the reality ofthe visible light image by modifying the image of the subject on thevisible light image in accordance with a change in a physical state ofthe subject.

3-1. Configuration of Second Embodiment

FIG. 6 exemplifies the configuration of an image processing apparatus ofa second embodiment. The image processing apparatus 20-2 has aninformation acquisition unit 21-2 and an effect processing unit 22-2.The effect processing unit 22-2 has an effect modification setting unit223 and an image modification unit 224.

Similarly to the information acquisition unit 21-1, the informationacquisition unit 21-2 performs the recognition process and the regionsegmentation process for the subject on the basis of the visible lightimage. The information acquisition unit 21-2 identifies what type ofentity the subject is in the recognition process and generates the typeinformation. The information acquisition unit 21-2 also identifies theimage region of the subject in the region segmentation process. Inaddition, the information acquisition unit 21-2 acquires the temperatureinformation on each subject from the temperature information, which isthe state information generated by a state information generation unit12, on the basis of a processing result of the region segmentationprocess. Furthermore, the information acquisition unit 21-2 may confirmthe consistency between the temperature information and the typeidentification result to enhance the accuracy of the recognitionprocess. The information acquisition unit 21-2 acquires the typeinformation on the subject and its region and temperature information onthe visible light image for each subject to output to the effectmodification setting unit 223 of the effect processing unit 22-2.

The effect modification setting unit 223 sets effect modification thatimproves the reality, in accordance with the type and temperature of thesubject on the basis of the type information and temperature informationon the subject. The effect modification setting unit 223 stores themodification control information in advance for each effect modificationthat can be carried out. The modification control information is controlinformation for modifying a subject image on the visible light imageinto an image in which a change in a physical state is produced inaccordance with the temperature of an entity indicated by the typeinformation. For example, control information to modify a subject imageinto an image depicting that atmospheric fluctuations are produced byheat, control information to modify a subject image into an imagedepicting that a change in a physical state between a solid phase and aliquid phase is produced and, for example, the water surface is frozen,and the like are stored as the modification control information. Theeffect modification setting unit 223 selects modification controlinformation according to the type and temperature of the subject.Furthermore, the effect modification setting unit 223 may change amodification level by adjusting the modification control information inaccordance with the temperature of the subject. For example, themodification level is changed such that the fluctuations grow as thetemperature rises. In addition, the modification level is changed suchthat the sense of freezing becomes higher as the temperature lowers. Theeffect modification setting unit 223 outputs the set modificationcontrol information to the image modification unit 224.

The image modification unit 224 performs a modification process on theimage region of the corresponding subject using the modification controlinformation set by the effect modification setting unit 223, therebyimproving the reality of the visible light image. The image region ofthe subject is identified by the above-described region segmentationprocess and the modification control information is set in accordancewith the type and temperature of the subject whose image region has beenidentified. Accordingly, the image modification unit 224 can perform themodification process on the image of the corresponding subject using themodification control information. The image modification unit 224generates the visible light image in which the image of the subject hasbeen modified in accordance with the type and temperature of the subjectto improve the reality, and records the generated visible light image toa recording medium or outputs the generated visible light image toexternal equipment such as a display apparatus or a recording apparatus.

3-2. Operation of Second Embodiment

FIG. 7 is a flowchart illustrating the operation of the secondembodiment. In step ST11, the effect processing unit 22-2 initializes aparameter value i indicating the subject on which the effect process isto be performed and specifies the parameter value i as “0” to proceed tostep ST12.

In step ST12, the effect processing unit 22-2 sets the number ofsubjects to the number of objects N. The effect processing unit 22-2sets the number of subjects identified by performing the recognitionprocess and the region segmentation process in the informationacquisition unit 21-2 to the number of objects N and proceeds to stepST13.

In step ST13, the effect processing unit 22-2 initializes themodification control information to put the modification controlinformation into a state in which the modification to the visible lightimage is not to be made and proceeds to step ST14.

In step ST14, the effect processing unit 22-2 identifies whether theparameter value i is smaller than the number of objects N. In a casewhere the parameter value i is smaller than the number of objects N, theeffect processing unit 22-2 proceeds to step ST15 and, in a case wherethe parameter value i is equal to or larger than the number of objectsN, the effect processing unit 22-2 terminates the process.

In step ST15, the effect processing unit 22-2 selects the modificationcontrol information. The effect modification setting unit 223 of theeffect processing unit 22-2 selects the modification control informationon the basis of the type of an i-th subject and the temperatureinformation thereon and proceeds to step ST16.

In step ST16, the effect processing unit 22-2 sets the modificationcontrol information. The effect modification setting unit 223 of theeffect processing unit 22-2 adjusts the modification control informationset in step ST15 on the basis of the temperature information on the i-thsubject such that the modification control information is set to amodification level according to the temperature and proceeds to stepST17.

In step ST17, the effect processing unit 22-2 performs an imagemodification process. The image modification unit 224 of the effectprocessing unit 22-2 performs the modification process on the image ofthe i-th subject on the visible light image using the modificationcontrol information set in step ST16 and proceeds to step ST18.

In step ST18, the effect processing unit 22-2 updates the parametervalue i to specify the parameter value i as “i=i+1” and returns to stepST14.

The effect processing unit 22-2 performs the process in the flowchartillustrated in FIG. 7 to perform the modification process on eachsubject identified by the information acquisition unit 21-2 andgenerates a visible light image with improved reality.

FIGS. 8A and 8B illustrate a case where the temperature is high as anoperation example of the second embodiment. For example, FIG. 8A is animage in which an outdoor subject (for example, a utility pole) OBcappears. In a case where the temperature information indicates that thetemperature of the subject OBc is high, the image modification unit 224performs a modification process on the image of the subject OBc todepict that atmospheric fluctuations are produced, on the basis of themodification control information set on the basis of the temperatureinformation and the like. Accordingly, as illustrated in FIG. 8B, theimage processing apparatus 20-2 can generate a visible light image withimproved reality by adding an effect expressing high temperature (forexample, fluctuations) to the visible light image.

FIGS. 9A and 9B illustrate a case where the temperature is low as anoperation example of the second embodiment. For example, FIG. 9A is animage obtained by imaging outdoors. In a case where the temperatureinformation indicates that the temperature of an outdoor subject (forexample, a lake surface) OBd is low, the image modification unit 224performs a modification process on the image of the subject OBd todepict that freezing occurs, on the basis of the modification controlinformation set on the basis of the temperature information and thelike. Accordingly, as illustrated in FIG. 9B, the image processingapparatus 20-2 can generate a visible light image with improved realityby adding an effect expressing low temperature (for example, depictionof freezing) to the visible light image.

As described thus far, according to the second embodiment, modificationis made on the image of the corresponding subject on the basis of themodification control information set in accordance with the type andtemperature of the subject in regard to the images of the subjects onthe visible light image. Accordingly, it is possible to generate avisible light image having the reality, in which the imaging object andthe imaging situation are reflected. In addition, effects are added onthe basis of the actual temperature information on the subject, suchthat it is possible to maintain a natural influence of the effects.

4. Third Embodiment

In a third embodiment, the state information is the temperatureinformation and the image processing apparatus improves the reality ofthe visible light image by performing color adjustment.

4-1. Configuration of Third Embodiment

FIG. 10 exemplifies the configuration of an image processing apparatusof the third embodiment. The image processing apparatus 20-3 has aninformation acquisition unit 21-3 and an effect processing unit 22-3.The effect processing unit 22-3 has an effect adjustment setting unit225 and a color adjustment unit 226.

The information acquisition unit 21-3 performs a recognition process anda region segmentation process for the subject on the basis of thevisible light image. The information acquisition unit 21-3 identifies asubject of interest in the recognition process. In addition, theinformation acquisition unit 21-3 acquires temperature informationcorresponding to the region of the subject of interest identified in theregion segmentation process from the temperature information, which isthe state information generated by a state information generation unit12. Furthermore, the information acquisition unit 21-3 outputs theacquired temperature information on the subject of interest to theeffect adjustment setting unit 225 of the effect processing unit 22-3.Note that the information acquisition unit 21-3 may output, to theeffect adjustment setting unit 225, temperature information indicatingthe average temperature value of the temperature image, the averagetemperature of a central region, a temperature having the widest regionof equal temperature, or the like.

The effect adjustment setting unit 225 sets the color adjustment (colortemperature conversion) that improves the reality on the basis of thetemperature information, which is the state information output from thestate information generation unit 12. The effect adjustment setting unit225 sets adjustment control information in accordance with thetemperature indicated by the temperature information. The adjustmentcontrol information is control information that adjusts the visiblelight image to a warm color system or a cool color system. The effectadjustment setting unit 225 outputs the set adjustment controlinformation to the color adjustment unit 226.

Using the adjustment control information set by the effect adjustmentsetting unit 225, the color adjustment unit 226 performs coloradjustment (color temperature conversion) on the visible light image inwhich white balance has been adjusted, thereby improving the reality ofthe visible light image. The color adjustment unit 226 generates thevisible light image on which color adjustment has been performed inaccordance with temperature to improve the reality, and records thegenerated visible light image to a recording medium or outputs thegenerated visible light image to external equipment such as a displayapparatus or a recording apparatus.

Note that the image processing apparatus 20-3 may extract thetemperature information for each subject as in the second embodiment andperform color adjustment for each subject on the basis of the extractedtemperature information to generate the visible light image withimproved reality.

4-2. Operation of Third Embodiment

FIGS. 11A and 11B illustrate a case where the temperature is low as anoperation example of the third embodiment. For example, FIG. 11A is animage obtained by imaging outdoors. In a case where the temperatureinformation indicates that the outdoor temperature is low, the coloradjustment unit 226 performs color adjustment on the visible light imageto apply a color of the cool color system thereto (color temperatureconversion to raise the color temperature) on the basis of theadjustment control information set on the basis of the temperatureinformation and the like. Accordingly, as illustrated in FIG. 11B, theimage processing apparatus 20-3 can generate a visible light image withimproved reality by adding an effect expressing low temperature to thevisible light image.

FIGS. 12A and 12B illustrate a case where the temperature is high as anoperation example of the third embodiment. For example, FIG. 12A is animage obtained by imaging outdoors. In a case where the temperatureinformation indicates that the outdoor temperature is high, the coloradjustment unit 226 performs color adjustment on the visible light imageto apply a color of the warm color system thereto (color temperatureconversion to lower the color temperature) on the basis of theadjustment control information set on the basis of the temperatureinformation and the like. Accordingly, as illustrated in FIG. 12B, theimage processing apparatus 20-3 can generate a visible light image withimproved reality by adding an effect expressing high temperature to thevisible light image.

As described thus far, according to the third embodiment, coloradjustment (color temperature adjustment) is performed on the visiblelight image in which white balance has been adjusted, on the basis ofthe adjustment control information set in accordance with temperature.Accordingly, it is possible to generate a visible light image having thereality, in which the imaging situation is reflected and the sense oftemperature is emphasized. In addition, effects are added on the basisof the actual temperature information, such that it is possible tomaintain a natural influence of the effects.

5. Fourth Embodiment

In a fourth embodiment, the state information is the sound fieldinformation and, the image processing apparatus improves the reality ofthe visible light image by superimposing an effect component imageaccording to the sound field information onto the visible light image.The image processing apparatus also modifies the subject image on thevisible light image in accordance with the sound field information toimprove the reality of the visible light image.

5-1. Configuration of Fourth Embodiment

FIG. 13 exemplifies the configuration of the fourth embodiment. An imageprocessing apparatus 20-4 has an information acquisition unit 21-4 andan effect processing unit 22-4. The effect processing unit 22-4 has animage generation/modification setting unit 227 and asuperimposition/modification unit 228.

The information acquisition unit 21-4 identifies the direction of thesound source and the type of the sound source from the sound fieldinformation generated by a state information generation unit 12. Thedirection of the sound source can be identified using, for example, thetechnology disclosed in the document “Shoji Makino, et al., “BlindSource Separation based on Independent Component Analysis”, theInstitute of Electronics, Information and Communication EngineersTechnical Report, EA2003-45, vol. 103, No. 129 (2003), pp. 17-24”. Inaddition, as for the type of the sound source, for example, thetechnology disclosed in the document “Katsutoshi Itoyama, Hiroshi Okuno,“Parameter Estimation of Virtual Sound Source for Musical InstrumentSound”, Research Report on Information Processing Society of Japan, 2013(2013)” is used. According to this technology, in regard to a musicalinstrument sound having noise and distortion when sound sourceseparation is performed, it is possible to estimate a parameter of avirtual musical instrument sound source for obtaining a clean musicalinstrument sound that does not include such noise and distortion, suchthat the type of the object can be estimated utilizing this parameter.The information acquisition unit 21-4 acquires the direction of thesound source and the type of the sound source to output to the imagegeneration/modification setting unit 227 of the effect processing unit22-4.

The image generation/modification setting unit 227 generates an effectcomponent image that improves the reality on the basis of the type ofthe sound source. The image generation/modification setting unit 227generates the effect component image using, for example, an elementimage according to the type of the sound source. As the element images,for example, element images indicating vibrations or the like areprepared for percussion instruments such that effects for clear andcrisp sound can be added. Element images indicating musical notes or thelike are prepared for wind instruments such that effects as if soundscome from the tube can be added. Element images indicating musicalscores or the like are prepared for stringed instruments such thateffects as if playing melodies can be added. In addition, the elementimage is not limited to musical instruments, but element imagesaccording to sounds output from vehicles, or the like may be prepared.For example, element images and the like indicating warning whistlemarks or the like may be prepared for a horn sound of a car.

Furthermore, the image generation/modification setting unit 227 maycontrol attributes of the effect component image, for example, thedimension and the number of element images, on the basis of the soundfield information. For example, as the sound has more intensity, theeffect component image generation unit 221 enlarges the size of elementimage or increases the number of element images. The imagegeneration/modification setting unit 227 outputs the generated effectcomponent image to the superimposition/modification unit 228.

In addition, the image generation/modification setting unit 227 may setthe effect modification to improve the reality in accordance with theintensity of the sound indicated by the sound field information. Theimage generation/modification setting unit 227 stores in advance themodification control information relating to a plurality of effectmodifications for the sound field information. The modification controlinformation is control information that modifies the subject image onthe visible light image on the basis of the type of the sound source,the intensity of the sound, and the like. For example, modificationcontrol information to modify a subject image into an image depictingthat a blurring effect is produced by sound, modification controlinformation to modify a subject image into an image depicting thatfluctuations are produced in a subject by sound, and the like are storedas the modification control information. The imagegeneration/modification setting unit 227 selects the modificationcontrol information according to the type of the sound source and thelike. Furthermore, the image generation/modification setting unit 227may change a modification level by adjusting the modification controlinformation in accordance with the intensity of the sound. The imagegeneration/modification setting unit 227 outputs the set modificationcontrol information to the superimposition/modification unit 228.

The superimposition/modification unit 228 performs a process on theimage region of the corresponding subject using the effect componentimage generated by the image generation/modification setting unit 227and the modification control information set by the imagegeneration/modification setting unit 227, thereby improving the realityof the visible light image. The superimposition/modification unit 228superimposes the effect component image generated by the imagegeneration/modification setting unit 227 onto the subject image of thesound source. In addition, the superimposition/modification unit 228performs modification processes according to the modification controlinformation on the subject image of the sound source and other subjectimages. The superimposition/modification unit 228 superimposes theeffect component image at the position of the sound source and performsimage modification, for example, on a subject image different from thatof the sound source, thereby generating a visible light image withimproved reality. In addition, the superimposition/modification unit 228records the generated visible light image to a recording medium oroutputs the generated visible light image to external equipment such asa display apparatus or a recording apparatus.

Note that the image generation/modification setting unit 227 may performonly one of the generation of the effect component image and the settingof the modification control information such that thesuperimposition/modification unit 228 either superimposes the effectcomponent image or modifies the visible light image, on the basis of theoutput from the image generation/modification setting unit 227.

5-2. Operation of Fourth Embodiment

FIGS. 14A and 14B exemplifies a visible light image in which the realityis improved using the effect component image. In FIG. 14A, a visiblelight image obtained by imaging players of a percussion instrument OBe,a wind instrument OBf, and a stringed instrument OBg is illustrated. Thesuperimposition/modification unit 228 superimposes an effect componentimage generated in accordance with the type of the sound source by theimage generation/modification setting unit 227, for example, at theposition of an image indicating the percussion instrument OBe. Inaddition, the superimposition/modification unit 228 superimposes aneffect component image generated in accordance with the type of thesound source by the image generation/modification setting unit 227 atthe position of an image indicating the wind instrument OBf.Furthermore, the superimposition/modification unit 228 superimposes aneffect component image generated in accordance with the type of thesound source by the image generation/modification setting unit 227 atthe position of an image indicating the stringed instrument OBg.Accordingly, as illustrated in FIG. 14B, the image processing apparatus20-4 can generate a visible light image with improved reality by addingan effect according to the type of the sound source to the visible lightimage. Note that, in the visible light image illustrated in FIG. 14B, aneffect component image EOBe indicating vibrations is superimposed at theposition of an image indicating the percussion instrument OBe, an effectcomponent image EOBf indicating musical notes is superimposed at theposition of an image indicating the wind instrument OBf, and an effectcomponent image EOBg indicating musical scores is superimposed at theposition of an image indicating the stringed instrument OBg.

FIGS. 15A and 15B exemplifies a visible light image in which the realityis improved using the effect component image and the modificationcontrol information. In FIG. 15A, a visible light image obtained byimaging a speaker and a listener together is illustrated. Using themodification control information set in accordance with the type of thesound source and the intensity of the sound by the imagegeneration/modification setting unit 227, thesuperimposition/modification unit 228 modifies the image of the listenerOBh, for example, into an image in which a blurring effect standing forvibrations is produced. In addition, the superimposition/modificationunit 228 superimposes an effect component image generated in accordancewith the type of the sound source by the image generation/modificationsetting unit 227 onto an image indicating the speaker (sound source)OBj. Accordingly, as illustrated in FIG. 15B, the image processingapparatus 20-4 can generate a visible light image with improved realityby adding an effect according to the intensity of the sound to thevisible light image. Note that, in the visible light image illustratedin FIG. 15B, the image of the listener OBh is altered to an image inwhich a blurring effect is produced and the effect component image EOBjindicating vibrations is superimposed at the position of an imageindicating the speaker OBj.

As described thus far, according to the fourth embodiment, the effectcomponent image generated in accordance with the type of the soundsource and the intensity of the sound is superimposed onto the image ofthe sound source in regard to the images of the subjects on the visiblelight image. Furthermore, modification is made on the image of a subjectlocated around the sound source on the basis of the modification controlinformation set in accordance with the intensity of the sound.Accordingly, it is possible to generate a visible light image having thereality, in which the imaging object and the imaging situation arereflected. In addition, effects are added on the basis of the actualsound field state, such that it is possible to maintain a naturalinfluence of the effects.

6. Other Embodiments

The first to third embodiments may be performed individually or incombination. For example, the first and second embodiments may becombined such that the effect component image is superimposed and animage is modified based on the modification control information as inthe fourth embodiment to generate a visible light image with improvedreality. In addition, in the fourth embodiment, the effect componentimage is superimposed and an image is modified based on of themodification control information, but only one of these operations maybe performed to generate a visible light image with improved reality.Furthermore, the first to fourth embodiments may be carried out incombination using the temperature information and the sound fieldinformation as the state information.

The series of processes described in the description can be executed byhardware, software, or a complex configuration of both. In the case ofexecuting the processes by software, a program recording a processingsequence is installed on a memory within a computer incorporated indedicated hardware and executed. Alternatively, it is possible toinstall and execute the program on a general-purpose computer capable ofexecuting various processes.

For example, the program can be recorded in advance on a hard disk, asolid state drive (SSD), or a read only memory (ROM) as a recordingmedium. Alternatively, the program can be temporarily or permanentlysaved and kept (recorded) on a removable recording medium such as aflexible disk, a compact disc read only memory (CD-ROM), a magnetooptical (MO) disk, a digital versatile disc (DVD), a Blu-Ray Disc (BD)(registered trademark), a magnetic disk, or a semiconductor memory card.Such a removable recording medium can be provided as so-called packagesoftware.

Furthermore, in addition to installing the program from a removablerecording medium on a computer, the program may be wirelessly or wiredlytransferred from a download site to a computer via a network such as alocal area network (LAN) or the Internet. In the computer, it ispossible to receive the program transferred in such a manner and toinstall the program on a recording medium such as a built-in hard disk.

Note that the effects described in the present description merely serveas examples and not construed to be limited. There may be an additionaleffect not described herein as well. Furthermore, the present technologyshould not be interpreted as being limited to the embodiments of theabove-described technology. The embodiments of this technology disclosethe present technology in the form of exemplification and it isself-evident that those skilled in the art can make modifications andsubstitutions of the embodiments without departing from the gist of thepresent technology. That is, in order to judge the gist of the presenttechnology, claims should be considered.

Furthermore, the image processing apparatus of the present technologycan have the following configuration.

(1) An image processing apparatus including:

an information acquisition unit that acquires state informationcorresponding to a subject on a visible light image from stateinformation indicating a subject state not indicated on the visiblelight image; and

an effect processing unit that performs an effect process on the visiblelight image on the basis of the state information acquired by theinformation acquisition unit.

(2) The image processing apparatus according to (1), in which

the state information is temperature information, and

the effect processing unit performs an effect process according totemperature.

(3) The image processing apparatus according to (2), in which

the information acquisition unit acquires type information on thesubject by performing a recognition process for the subject, and

the effect processing unit superimposes an effect component imageaccording to a temperature and a type of the subject onto an image ofthe subject on the basis of the state information and the typeinformation.

(4) The image processing apparatus according to (3), in which the effectprocessing unit uses, as the effect component image, an imageschematically representing a visually observable phenomenon that occursin accordance with temperature.

(5) The image processing apparatus according to (3) or (4), in which theeffect processing unit controls an attribute of the effect componentimage in accordance with a difference in temperature of the subject.

(6) The image processing apparatus according to any one of (2) to (5),in which the effect processing unit modifies a subject image on thevisible light image into an image in which a physical state of thesubject is changed in accordance with a temperature of the subject.

(7) The image processing apparatus according to (6), in which the effectprocessing unit regards a change between a solid phase and a liquidphase as the change in the physical state.

(8) The image processing apparatus according to (6) or (7), in which theeffect processing unit controls a modification level in accordance witha difference in the temperature.

(9) The image processing apparatus according to any one of (2) to (8),in which the effect processing unit performs color adjustment on thevisible light image in accordance with a temperature of the subject.

(10) The image processing apparatus according to (9), in which theeffect processing unit performs color adjustment to a warm color systemwhen a temperature of the subject is high and performs color adjustmentto a cool color system in a case where a temperature of the subject islow.

(11) The image processing apparatus according to any one of (1) to (10),in which

the state information is sound field information, and

the effect processing unit performs an effect process according to thesound field information.

(12) The image processing apparatus according to (11), in which theeffect processing unit superimposes an effect component image onto asubject on the visible light image corresponding to a position of asound source based on the sound field information.

(13) The image processing apparatus according to (12), in which theeffect processing unit superimposes an effect component image associatedwith a sound output from the sound source onto a region of the visiblelight image.

(14) The image processing apparatus according to any one of (11) to(13), in which the effect processing unit adjusts an effect amount forthe visible light image in accordance with intensity of a sound outputfrom a sound source based on the sound field information.

INDUSTRIAL APPLICABILITY

In the image processing apparatus, the image processing method, and theprogram according to this technology, state information corresponding toa subject on a visible light image is acquired from state informationindicating a subject state not indicated on the visible light image andan effect process is performed on the visible light image on the basisof the acquired state information. Therefore, the improvement of thereality can be achieved in the visible light image. Accordingly, it issuitable for an imaging apparatus, an editing apparatus, and the like.

REFERENCE SIGNS LIST

-   10 System-   11 Visible light image generation unit-   12 State information generation unit-   20, 20-1, 20-2, 20-3, 20-4 Image processing apparatus-   21, 21-1, 21-2, 21-3, 21-4 Information acquisition unit-   22, 22-1, 22-2, 22-3, 22-4 Effect processing unit-   221 Effect component image generation unit-   222 Effect component image superimposition unit-   223 Effect modification setting unit-   224 Image modification unit-   225 Effect adjustment setting unit-   226 Color adjustment unit-   227 Image generation/modification setting unit-   228 Superimposition/modification unit

The invention claimed is:
 1. An image processing apparatus, comprising:circuitry configured to: acquire state information corresponding to asubject on a visible light image, wherein the state informationcorresponds to temperature information indicating a temperature of thesubject, and an indication of the temperature of the subject is absentin the visible light image; execute a recognition process for thesubject; acquire type information of the subject based on the executionof the recognition process, wherein the type information indicates atype of the subject; and superimpose an effect component image on thesubject in the visible light image based on the temperature informationindicating the temperature of the subject, and the type informationindicating the type of the subject.
 2. The image processing apparatusaccording to claim 1, wherein the effect component image represents avisually observable phenomenon based on the temperature information. 3.The image processing apparatus according to claim 1, wherein thecircuitry is further configured to control an attribute of the effectcomponent image based on change in the temperature of the subject. 4.The image processing apparatus according to claim 1, wherein thecircuitry is further configured to: modify a subject image of thesubject on the visible light image into a specific image; and change aphysical state of the subject in the specific image based on thetemperature of the subject.
 5. The image processing apparatus accordingto claim 4, wherein the change in the physical state corresponds to achange between a solid phase of the subject and a liquid phase of thesubject.
 6. An image processing method, comprising: acquiring, bycircuitry, state information corresponding to a subject on a visiblelight image, wherein the state information corresponds to temperatureinformation indicating a temperature of the subject, and an indicationof the temperature of the subject is absent in the visible light image;executing, by the circuitry, a recognition process for the subject;acquiring, by the circuitry, type information of the subject based onthe execution of the recognition process, wherein the type informationindicates a type of the subject; and superimposing, by the circuitry, aneffect component image on the subject in the visible light image basedon the temperature information indicating the temperature of thesubject, and the type information indicating the type of the subject. 7.A non-transitory computer-readable medium having stored thereon,computer-executable instructions which when executed by a computer,cause the computer to execute operations, the operations comprising:acquiring state information corresponding to a subject on a visiblelight image, wherein the state information corresponds to temperatureinformation indicating a temperature of the subject, and an indicationof the temperature of the subject is absent in the visible light image;executing a recognition process for the subject; acquiring typeinformation of the subject based on the execution of the recognitionprocess, wherein the type information indicates a type of the subject;and superimposing an effect component image on the subject in thevisible light image based on the temperature information indicating thetemperature of the subject, and the type information indicating the typeof the subject.
 8. An image processing apparatus, comprising: circuitryconfigured to: acquire state information corresponding to a subject on avisible light image, wherein the state information corresponds totemperature information indicating a temperature of the subject, and anindication of the temperature of the subject is absent in the visiblelight image; modify a subject image of the subject on the visible lightimage into a specific image; and change a physical state of the subjectin the specific image based on the temperature of the subject, whereinthe change in the physical state corresponds to a change between a solidphase of the subject and a liquid phase of the subject.
 9. An imageprocessing method, comprising: acquiring, by circuitry, stateinformation corresponding to a subject on a visible light image, whereinthe state information corresponds to temperature information indicatinga temperature of the subject, and an indication of the temperature ofthe subject is absent in the visible light image; modify a subject imageof the subject on the visible light image into a specific image; andchange a physical state of the subject in the specific image based onthe temperature of the subject, wherein the change in the physical statecorresponds to a change between a solid phase of the subject and aliquid phase of the subject.